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@ A method for detecting a countour of a given 
subject to be separated from images and an image 
separating system. First, a plurality of candidate 
points (110...120) which have been input by a user 
are displayed on a screen. A smooth curve (100) 
which connects the plurality of candidate points is 
determined and is displayed on a display apparatus 
(Fig. 4A). the intensity of light on a straight line 
(130) extending in a direction normal to each can- 
didate point is determined (Fig. 48), The position of 
the candidate point is corrected based upon the rate 
of change in the intensity of light which is deter- 
mined for each candidate point. A smooth curve 
(101, 102) which passes the corrected points is 
determined and is displayed on the display appara- 
tus (Figs. 4C. 4D). 
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1 EP 0 532 

BACKGROUND OF THE INVENTION 

The present invention relates to an image-sep- 
arating point detecting process for separating only 
a subject image from images having gradation and s 
an image separating apparatus using the same and 
. in particular to a method for detecting points to 
separate a specific subject image from a multi- 
color natural images which are picked up by a 
television camera and an image separating system. to 

An image synthesizing technique for merging a 
given subject image in one scene into a different 
other background scene and the like has heretofore 
been used in the. process for preparation of a 
picture such as movie or commercial poster. The 75 
commercial image synthesizing technique includes 
two approaches as follows: 

(a) A screen projection approach. An image 
which will become a background is projected on 

a screen and a given subject or object is placed 20 
before the screen and the image of the given 
subject is picked up together with the back- 
ground, 

(b) A chromakey synthesizing approach. A sin- 
gle background (usually referred to as blue 2S 
back) is used at the time of editing. The image 

. of a given subject is extracted on the basis of 
the difference in color components of the given 
subject and the background and is merged into 
the other image (background). In both methods, 30 
a special apparatus has to be used on picture 
taking. It is hard to merge an image which has 
previously been picked up with the other dif- 
ferent background scene. 
In other words, in order to achieve image syn- .35 
thesizing for an image which has been picked up, 
this synthesizing should be carried out in the ed- 
iting phase. 

Recently, home video cameras have become 
popular. There may be a need to remove a redun- 40 
dant image from a number of images which were 
picked up in festival or travel, or to conduct edition 
such as synthesizing the image of a person with a 
special background. Also in this case, enormous 
labour is necessary since such working is manually 45 
carried out for each of images if the image is a 
motion picture. 

As one of editing functions, an operation to 
automatically extract a given subject from taken 
images is required. 50 
(a) A method for separating a given subject 
image which is performed by a computer has 
been proposed in. for example. "SNAKES: Ac- 
tive Contour Models'* (Proc. ICCV-87, PP. 259 to 
268, 1987). The method comprises modelling 55 
the contour of a subject based upon the f'eatures 
of the subject, defining the contour as a function 
and inferring the contour in the vicinity of an 
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initial input by determining the contour in accor- 
dance with the features of the preliminarily de- 
fined function even if a part of the contour of the 
subject is not clear. 
. (b) A technique to separate a motion subject by 
determining differences between the frames of 
motion picture is disclosed in, for example, 
"Development of Video Surveillance System us- 
ing Motion Information" (October National Con- 
vention Summary Record of the Institute of 
Electronics and Communication Engineers of Ja- 
pan, page 7-178, -1991). First, differences be- 
tween frames are determined by performing a 
linear operation for each pixel of the image over 
continuous frames. Thereafter, the points of 
changes are used to trace a locus of a moving 
subject for separating a given subject image, 
(c) A system for separating a given subject 
image by using color information of image is 
disclosed in, for example, JP-B-2-4027. In this 
system, a histogram is determined by perform- 
ing a color linear operation for each pixel and 
then a given subject image is separated by 
thresholding the results. 

SUMMARY OF THE INVENTION 

The methods (a) and (c) have not been prac- 
tically used for automatically extracting a desired 
specific subject for the reasons as follows: The 
separation accuracy is insufficient when they are 
applied for a natural image by merely performing a 
functional approximation based upon a simple 
model or a linear operation for each pixel since the 
conditions and the environment are various, for 
example, the complication of the shape of the 
contour of one subject is different from those of the 
other subjects, a local deviation of the. distribution 
of the intensity of light may occur on a specific 
subject by reflections of light within the contour 
depending on the way of lighting, or a given sub- 
ject of natural images may often include complex 
hues. In the method (b) for determining differences 
between frames, only a specific subject cannot be 
detected at a high accuracy since a background or 
the other background included in a recess may 
also be detected if the specific subject includes the 
recess. 

It is therefore an object of the present invention 
to provide an image separating points detecting 
method and an image separating system which 
overcomes the above mentioned problems of the 
prior art and are capable of precisely extracting a 
given subject in images and detecting separating 
spots. 

In order to accomplish the above mentioned 
object, in an aspect of the present invention, there 
is provided a method for detecting image-separat- 
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ing points where only a given subject image is 
separated from images, comprising the steps of: a 
first step for firstly displaying on a screen a plural- 
ity of candidate points in the vicinity of a boundary 
of the given subject image, which are inputted by a 
user; a second step for determining and displaying 
a smooth curve which connects the plurality of 
candidate points; a third step for determining the 
intensity of light on a straight line normal to the 
curve in- each of the plurality of the candidate 
points; a fourth step for correcting the positions of 
the candidate points based upon the rate of change 
in the intensity of light determined in each of the 
candidate points; and a fifth step for determining 
and displaying a smooth curve passing the cor- 
rected candidate points. The method preferably 
includes the steps of: individually processing the 
same image in each of components having dif- 
ferent wave lengths by using the third and fourth 
steps; determining the positions of the corrected 
candidate points for each component; calculating 
the position of one corrected candidate point from 
a plurality of corrected candidate points after pro- 
cessing of said fourth step; performing the sixth 
step for determining the final position; and there- 
after performing the fifth step. When the rate of 
change in the intensity of light is determined in the 
fourth step, the designated values of the threshold 
parameters for determining the rate of change for 
detecting an edge are changed. When a smooth 
curve connecting the plurality of candidate points is 
determined in the second step, the position of at 
least one of the candidate points is preliminarily 
fixed and processing in the third and fourth steps is 
stopped for the fixed candidate point, while pro- 
cessing of the second to fifth steps is performed 
for the other candidate points. In the fourth step, 
processing for determining the rate of change in 
the intensity of light is sequentially performed from 
the external to internal* slide in each candidate point 
on a coarse curve traced in the vicinity of the 
contour of the subject and a point which satisfies 
the conditional formulae representing that the rate 
of change for correcting the plurality of candidate 
points is higher than a threshold is determined as 
the position of the corrected candidate point. After 
processing of the fifth step, the threshold of the 
rate of change is desirably changed for at least one 
candidate point in which a point which satisfies the 
conditional formulae as a result of the determina- 
tion of the rate of change in light intensity in the 
fourth step and then the processing of the third to 
fifth steps is repeated at least once. If processing is 
repeated again from the second step after comple- 
tion of the fifth step, at least one candidate point is 
fixed and the third and fourth steps are stopped for 
the fixed candidate point and each processing of 
the third, fourth and fifth steps is repeated at least 



once from a point adjacent to selected point for the 
other candidate points. The process preferably fur- 
ther includes determining a candidate point input- 
ted in the first step be located in the position of 

5 corrected candidate point; and adding the can- 
didate point with an attribute representing that It 
does not satisfy the conditional formulae for cor- 
recting the position, including displaying the can- 
didate point in a color different from that of the 

70 other point on the screen. The process preferably 
further includes determining a candidate point in- 
putted in the fourth step be located in the position 
of the corrected candidate point; and adding the 
candidate point with an attribute representing that it 

75 does not satisfy the conditional formulae for cor- 
recting the position, including displaying the can- 
didate point in a color different from that of the 
other point on the screen. In another aspect of the 
present invention, there is provided an image sepa- 

20 rating system for separating a given subject image 
from images, comprising: means for determining a 
coarse contour curve for the given subject image 
which is desired to be separated; means for deter- 
mining a plurality of candidate points on said con- 

25 tour curve; means for correcting the position of 
each of said candidate points in each spectral 
component based upon the rate of change in light 
intensity of each spectral component; means for 
comparing the candidate points corrected in each 

30 spectral component with each other; means for 
specifying separating points from the results of 
comparing; and means for separating the image 
from the specified points. 

The. present invention provides steps for. se- 

35 iecting the position of a contour of a given subject 
image which seems to be most accurate based 
upon the conditions and environment by inputting a 
plurality of candidate points in the vicinity of a 
boundary between the given subjective image and 

40 a background. 

In other words, (1) a plurality of points for 
detecting the edge shape in a subject are firstly 
specified. (2) Then, a user specifies plural points 
among the specified plural points above for detect- 

45 ing the edge including at least one candidate point, 
changes the parameters for determining the edge 
positions in these points, or fixes the edge posi- 
tions. (3) A smooth curve passing the candidate 
points for .inferring the direction of the edge to be 

50 detected is determined. (4) The edge position is 
determined based upon the intensity of light of the 
image and the rate of change thereof in a direction 
normal to the curve in each candidate point is 
determined. In case of multi-spectral image, the 

55 edge position is determined for each of red. green 
and blue components. (5) The error of the position 
of the edge of each color component is corrected 
in accordance with the degree of the rate of 
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change in light intensity and the candidate points 
are placed on the corrected positions of the edge 
and the original candidate points are left un- 
changed for the points where the edge cannot be 
detected or corrected or for the points the positions 
of which are preliminarily fixed. (6) After completion 
of a series of the processings, the user specifies 
one or more candidate points, changes the param- 
eters for determining the edge position in the 
points or fixes the edge position, and detects the 
edge so that the contour of the specific subject to 
be separated is deformed by placing the candidate 
points on the corrected edge positions. 

In such a manner, a contour can be automati- 
cally determined by successively determining the 
edge of a given subject to be separated using the 
most suitable color information in each candidate 
point. It is possible to determine the candidate 
points depending upon various conditions and envi- 
ronment which are specific for the natural image by 
specifying the location where it is hard to deter- 
mine the contour and by changing the parameters. 
Although the contour of the subject is finally deter- 
mined along the entire circumference thereof in 
such a method, the contour, the shape of which is 
so complicated that the accuracy of the separation 
of the contour by the automatic separating pro- 
cessing seems to become adverse is manually 
determined by an interactive processing. Since 
separation of the contour having a simple shape 
can be completely automated, a load upon an 
operator can be reduced and a given subject sur- 
rounded by the contour line can be extracted at a 
high accuracy. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram showing an embodi- 
ment of an image editing system of the present 
invention; 

Fig. 2 is a block diagram showing an embodi- 
ment of an image separating system of the 
present invention; 

Fig. 3 is a flow chart showing a process for 
detecting image separating positions of the 
present invention; 

Figs. 4A through 4D are views which are useful 
for illustrating the distribution of the intensity of 
light in a normal direction of a contour curve and 
a process of detecting edges of the present 
invention; 

Figs. 5A through 5C are views useful for explain- 
ing a process for determining the edge position 
from the light intensity and the rate of change 
thereof for each color component in the, present 
invention; 

Figs. 6A, 6B. 7A and 78 are views useful for 
explaining a process for determining the edge 



position from each color component, light inten- 
sity and the change rate which meet the thresh- 
old conditions; 

Figs. 8A through 8C are flow charts showing an 
5 operation which is performed by using the edge 

positions determined from a plurality of color 
components; 

Figs. 9A through 9C are views illustrating an 
operation method performed in a system for 

70 determining the image separating points in ac- 
cordance with the present invention; 
Figs. 10A through lOC are views illustrating an- 
other operation method performed in the system 
for determining the image separating points in 

75 accordance with the present invention; 

. Figs. 11A through 11C are views illustrating an- 
other operation method performed in the system 
for determining the image separating points in 
accordance with the present invention; 

20 Figs. 12A through 12C are views illustrating an- 
other operation method performed in the system 
for determining the image separating points in - 
accordance with the present invention; 
Figs. ISA through 13C are views illustrating an- 

25 other operation method performed in the system 
-for determining the image separating points in 
accordance with the present invention; and 
Fig. 14 is a block diagram showing an embodi- 
ment of an image separating system of the 

30 present invention. 

DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

35 Embodiments of the present invention will now 

be described with reference to the drawings. 

Fig. 1 is a view showing the whole structure of 
an image editing system of the present invention. 
In Fig. 1, reference numeral 10 denotes an image 

40 processing apparatus; 1 1 an image display appara- 
tus for displaying an image which is being pro- 
cessed or a result of processing; 12 a memory unit 
having a high capacity for storing images: 13 a 
pointing device for designating the coordinate in 

45 the displayed image; 14 an image memory in- 
cluded in the memory unit 12 for storing a two- 
dimensional digital image: 15 an image recording 
apparatus in which a number of images picked up 
by a television camera and the like are stored; and 

50 16 a separating memory included in the memory 
unit 12 similarly to the memory 14 for storing a 
result of processing in the image processing ap- 
paratus. 

Image data which are sequentially read from 
55 the image recording apparatus 15 are stored in the 
image memory 14 as digital data. A user instructs 
the image processing apparatus to perform the 
designation of the coordinates in candidate points 



4 



7 



EP 0 532 823 A2 



8 



and different processing in predetermined candi- 
date points by inputting positions by using the 
pointing device 13. Positioning of processing for 
detecting the image separating points can be 
achieved by an intuitive operation to directly des- 
ignate the positions in the displayed image by 
using the pointing device 13. 

The user firstly reads first image from the 
image memory 14 and inputs the coarse contour of 
an image which the user desires to separate while 
looking at the position on the image display ap- 
paratus 11. The image processing apparatus 10 will 
then commence an automatic operation- to perform 
the image separating point detecting processing on 
the basis of given coarse contour information. After 
the separation of a subject from the two dimen- 
sional digital image which is read from the image 
memory 14 is completed, the image processing 
apparatus 10 stores the result in the separating 
memory 16. The subject separating step deter- 
mines each coordinate on the external contour line 
of the image to be separated by interpolating the 
image separating points which are a set of the 
determined separating points with a linear line or a 
smooth curve. Then the image inside or outside of 
the external line is then separated on the basis of 
these coordinates. The separating processing can 
be easily reali2ed by using a method which is 
described in detail in, for example, an article 
"Scanline Conversion of a Polygon (2)" PIXEL No. 
16, pp. 136-141, Jan. 1984. 

The present embodiment performs the image 
separating point detecting and image separating 
process and is not limited only the above men- 
tioned method. 

The user can designate the separating points 
in the displayed image and cope with the image 
conditions and environmental versatility. 

Fig. 2 is a block diagram showing the whole of 
another embodiment of an image separating sys- 
tem of the present invention. 

The embodiment of Fig. 2 is substantially iden- 
tical with that of Fig. 1 except that the memory unit 
18 includes three memories such as an image 
memory 14, an separating memory 16 and a con- 
tour memory 17 in the embodiment of Fig. 2 while 
the memory unit 18 includes two memories such 
as the image memory 14 and the separating mem- 
ory 16 in the embodiment of Fig. 1 and that a 
coarse contour is temporarily stored in the memory 
unit 18 in Rg. 2 while it is temporarily stored in a 
buffer memory in the image processing apparatus 
10. 

The contour memory 17 holds the values of 
coordinates of the coarse contour. In other words, 
the contour memory 17 is an alternative area for 
inputting the coarse contour which has been pro- 
vided by using the pointing device 13 in Fig. 1. 



These coarse coordinate values are prepared by 
using the image processing apparatus 10 for the 
image read out from the image memory 14. A 
linear operation among pixels which is a prior art 
5 method can be used for extracting a coarse con- 
tour. 

Since use of such an arrangement makes it 
possible to automatically provide the image pro- 
cessing apparatus with the coarse contour which 
10 was determined by the previous processing, the 
operation imposed upon the user for a large quan- 
tity of image such as movie image can be reduced, 
and the image can be separated at a high preci- 
sion. 

75 Fig. 3 is a flow chart showing an operation of 

the image separating point detecting method in 
accordance with the present invention. Figs. 4A 
through 4D are views illustrating the operation for 
detecting the image separating points with refer- 

20 ence to a mono-color gray level image for easy 
understanding of a color image. Figs. 4A through 
4D show the distribution of the light intensity in a 
normal direction of a contour curve and edge de- 
tection in the candidate points when the image 

25 separating points are detected and changes in 
spline curve based upon the detection results in 
the corrected candidate points. 

It is a final goal of the image separating point 
detection to determine the contour 102 of a given 

30 subject image as shown in Fig. 40 at a high 
precision and to separate the subject image 40 on 
the basis of this. 

To this end, after the whole of an image is 
firstly displayed on the image display apparatus 1 1 , 

35 the candidate points of a subject image 40 are 
designated by using the pointing device 13 and 
coordinates of the candidate points are inputted 
into the image processing apparatus 10 for display- 
ing them on the display apparatus as shown in Fig. 

40 3 (step 21). For example, a part of the contour of 
the subject image 40 and 1 1 initial candidate points 
110 to 120 are designated and these coordinates 
are inputted as shown in Fig. 4A. Then, a spline 
curve 100 passing through the candidate points 

45 110 to 120 is determined as shown in Fig. 4A (step 
26). The curve 100 is traced to pass through a 
plurality of positions which are firstly designated. 
The shape of the curve is changed by being con- 
secutively corrected as will be described hereafter. 

50 For a portion of the curve which the user deter- 
mines that it is difficult to perform detection and 
processing in view of the state of the image, the 
initial candidate points are fixed by using the point- 
ing device 13 according to needs (steps 22 and 23) 

55 or parameters of the separating points detection 
are changed (steps 24 and 25). Operation of 
changing the parameters specifically includes des- 
ignating the candidate points where the thresholds 
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of the edge detection are changed. In order to 
perform the additional designation, it is convenient 
to use a pop-up type menu (for example, a display 
illustrated in the left and upper side of the screen 
in Fig. 9A) which is used in the graphical user 
interface, etc. 

When the curve is consecutively corrected to 
change its shape, determination is first made for 
each candidate point. Determination whether or not 
a candidate point is to be subject to movement 
(step 31). If there is no candidate point to be 
subject to movement, the desired curve 102 as 
shown in fig. 4D has been obtained. Accordingly, 
the processing is terminated. If there are some 
candidate points to be subject to movement, cor- 
rection as shown in figs. 4A to 4C is made. A graph 

140 in Fig. 4B shows the relation between the 
normal vector 130 passing through the candidate 
point 113 on the contour in Fig. 4A and the inten- 
sity of light of an image through which the normal 
vector 130 passes wherein the normal vector is 
represented in abscissa and the light intensity is 
represented in ordinate. At this time a difference 
between the intensity of light in the position of the 
candidate point 113 and that in the vicinity thereof 
is determined (step 27). In other words, the inten- 
sity of light of the subject image 40 in a direction 
from an external position to internal position along 
the normal vector 130 passing through the can- 
didate point 113 of the spline curve 100 passing 
through the initial candidate points is determined. 
Fig. 4B shows the relation between the coordinates 
of the light intensity and the normal vector. In Fig. 
4B, an edge is detected from the rate of change in 
light intensity in coordinates in a given region along 
the normal vector 130 and is determined as an 
image separating candidate point. The determina- 
tion condition of the edge satisfies three formulae 
as follows: 

|la-lb|<Tave (1) 
|lc - ld| < Tave (2) 
|lb - lc| > Tedge (3) 

where Tave denotes the threshold representative of 
the smoothness of the light intensity of the image; 
Tedge is the threshold representative of the rate of 
change in the light intensity; and a, b, c, d are 
pixels which are continuous on the normal vector 
130 and Ix (x: a, b, c, d) is the light intensity of 
each pixel. |a| denotes the absolute value of a. 

In Fig. 4B. a portion represented by a line 141 
satisfies the conditions (1) to (3). The light intensity 
in the left side of a point corresponding to the line 

141 is remarkably higher than that in the right side 
of the line 141. A value which is higher than the 



threshold representative of the smoothness of the 
fight intensity and is higher than the threshold 
representative of the rate of change of the light 
intensity exists in a position between the higher 

5 level left side and the lower level right side. This 
position is the boundary of the subject, i.e. the 
position of a point 121. In order to detect an image 
separating point, the processing proceeds from the 
external area to internal area of the subject image 

10 40 along the normal vector 130 as shown in Fig. 3. 
A first detected edge is made a candidate point 
121 of the image separating point (step 32). Then, 
the candidate point 121 is made a new candidate 
point and a new spline curve 101 passing through 

75 this position is defined and is displayed on the 
display apparatus (step 33) as shown in Fig. 4C. 
Then, processing is performed along a normal vec- 
tor 131 passing through the adjacent candidate 
point 114 as is similar to processing for the normal 

20 vector 131. 

If the image is a multi-color image, this pro- 
cessing is performed for each spectral component 
(step 28) as shown in Figs. 5A to SC. If the image 
is a mono-color gray image, step 28 is omitted. 

25 The above mentioned processing is consecu- 

tively repeated for the candidate points 110 to 120 
as represented by a spline curve 102 by repeating 
a return loop from step 29 of termination process- 
ing to the step 22. Thus, new candidate points 122 

30 to 125 are determined and the spline curve is 
updated on the basis of these candidate points for 
calculating the candidate points of the image sepa- 
rating points. 

The candidate points 110 to 112, 116. 117 and 

35 120 are those initial candidate points that match 
with new candidate points. These points may be 
preliminarily designated by the pointing device 13 
excepting the state of the subject image 40, Simi- 
larly, the threshold values Tave and Tedge in the 

40 right terms of the formulae (1) to (3) may be 
designated as different values by preliminarily des- 
ignating some of the initial candidate points so that 
the rate of change in the light intensity of a part of 
the subject image 40 is easily processed. 

45 Figs. 5A to 5C are graphs explaining step 28 in 

Fig. 3. The relation between the edge detection for 
determining the positions of the candidate points 
for separating the contour of a multi-color image 
and the searching direction is illustrated. 

50 In Figs. 5A to 5C, the relation between the light 

intensity of each color component and the coordi- 
nate on the norma! vector of a spline function 
passing the vicinity of the contour is shown as is 
similar to Fig, 4B. Fig. 5A shows a case in which all 

55 candidate points Er, Eg and Eb for separating the 
contour based. upon the mono-color gray image of 
each of red, green and blue color light intensities 
200, 201 and 202 exist in a range of a given length 
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^L. In this case, the position Pi* (2n in Fig. 5A) of 
the candidate point which Is corrected for contour 
separating is the position which is moved to an 
average coordinate of three candidate points from 
the initial candidate point 210. Fig. 5B shows a 
case in which two candidate points Er and Eb of 
the red and blue color intensities 203 and 205 exist 
in a given length AL and only the candidate point 
Eg of the green color intensity exists on side of this 
range. In this case, the position of the candidate 
point which is corrected for separating contour is 
the position which is moved to an average coordi- 
nate 213 of the two candidate points Er and Eb in 
the given length AL from the initial candidate point 
212. Fig. 5C show a case in which none of the 
candidate points Er, Eg and Eb of the red. green 
and blue color light intensities 207, 208 and 206 
exists in a given length AL. In this case, the' posi- 
tion Pi* (215 in Fig, 5C) of the corrected candidate 
points is a value of the initial candidate point Pi 
(214 in Fig. 5C). 

Figs. 6A, 6B and 7A, 7B show a method for 
determining the positions of the candidate points 
which are corrected for separating the contour 
when the edge of some or any of each color 
component is not determined. Figs. 8A, SB and 8C 
are flow charts showing a process for detecting the 
positions and calculating the coordinates of the 
candidate points which are corrected for separating 
the contour in the color image illustrated in step 28 
in Fig, 3. 

In Fig, 8A. the candidate point of the contour 
separation in each of three colors is detected (step 
28a) and three points are in a predetermined range 
(step 28b). Accordingly, an average coordinate of 
three points is determined (step 28c). The can- 
didate point in three colors is detected in case of 
Fig. 58 (step 28a). None of three points is close to 
each other (step 28b). Only the points of the red 
and blue color light components are dose to each 
other. Accordingly, an average coordinate of the 
two points is obtained (step 28f ). The candidate 
points in three colors are detected in case of Fig. 
5C (step 28a). Since any of the three points is not 
close to each other (steps 28b and 28c). detection 
of the corrected position is impossible (step 28y). 
At this time, a first designated coordinate is se- 
lected (step 28h). 

Figs. 6A. 6B and 7A, 78 show a method for 
determining the corrected candidate points if can- 
didate points cannot be selected in a part or any of 
the mono-color gray image. Figs. 6A and 6B show 
a case where a candidate point of two components 
is determined. In other words. Fig. 6A shows a 
case where the determined candidate points are 
within a range of a given length AL. At, this time, 
the position Pi* (232 in Fig. 6A) of the candidate 
point which is corrected for the contour separation 



of the initial candidate point 23i is an average 
coordinate of two candidate points Er and Eg of the 
red and green color light intensities 221 and 222. In 
a method of Fig. 8A corresponding to Rg. 6A. the 

5 candidate points of three colors are not detected, 
but the candidate points of two colors are detected 
(step 28d). Since these two points are close to 
each other (step 28e). an average coordinate of the 
two points is obtained (28f). 

70 Fig. 6B shows a case where two candidate 

points, i.e. the candidate points Eg and Eb of the 
green and blue color light intensities 225 and 223 
are not in a range of a given length AL. In this 
case, the position Pi (234 of Fig. 6) of the corrected 

75 candidate point is the coordinate of the initial can- 
didate point Pi (233). In processing of Fig. 8A, two 
points of two colors are detected, but are not close 
to each other (step 28e). Accordingly, it is impos- 
sible to detect the corrected position (step 28y). 

20 The corrected position is the original designated 
coordinate (step 28h). 

Fig. 7A shows a case where the candidate 
point of only one component is determined. In this 
case, the position Pi* (236 in Fig. 7A) of the can- 

25 didate point corrected for the contour separation 
assumes the coordinate of the candidate point Er 
which is determined by the red color light intensity 
in position of the initial candidate point 235. In 
other words, since the position of one color is 

30 detected in processing of Fig. 8A (step 28g), the 
position of the candidate point is the coordinate in 
the detected point (step 28i). 

When no candidate point is obtained in all 
components of red, green and blue color light 

35 intensities 228, 229 and 230 as shown in Fig. 7B. 
the position Pi* (238 in Fig. 7B) of the corrected 
candidate point is the initial candidate point Pi 
(237). Since the corrected candidate point is not 
detected in three colors (step 28a). is not detected 

40 in two colors (step 28d) and is not detected in one 
color (step 28g) and it is impossible to detect the 
corrected position (28y) in Fig. 8A, the corrected 
position is the initially designated coordinate (step 
28h). 

45 An example is described with reference to pri- 

mary color components. Combinations of the color 
components may be different if the contents of the 
color components are independent of each other. 
When detection of the candidate point is impos- 

50 sible in any case, the candidate point is the initial 
candidate point Pi of the original designated coordi- 
nate (step 28h in Fig. 8 A). The length between 
coordinates is determined from AL in Figs, 5A to 
5C. 

55 In Figs. 8B and 8C, the certainty when the 

candidate point is evaluated by a value (magnitude 
of contrast) represented in the left term of the 
formula (3) in addition to information whether or not 
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the candidate point in each color connponent is 
obtained. In Fig. 8B. determination whether in how 
nriany colors of three colors the correction position 
of the candidate point can be detected is naade in 
accordance with this evaluation (step 28i). If no 
position is detected, it is determined that detection 
of the corrected position of the candidate point is 
impossible (step 282) and the firstly given can- 
didate point is outputted (step 28p). If the can- 
didate positions can be detected in some colors, 
determination is made whether or not the corrected 
position of the candidate point determined in each 
color is in the range of a predetermined length AL 
(step 28k). If all of them are in the range of a 
predetermined length AL, an average coordinate of 
the corrected positions of the candidate points de- 
termined in these colors is outputted as a new 
candidate point (step 281). On the other hand, if 
the corrected position of the candidate point deter- 
mined in each color is not in the range of a 
predetermined length AL, the magnitude of con- 
trast (rate of change in light intensity) which is a 
measure for detecting the corrected position of the 
candidate point for each detected color is deter- 
mined (step 28m). Determination is made whether 
or not the highest contrast is higher than threshold 
representative of a predetermined magnitude of the 
rate of change in light intensity (step 28n). If it is 
lower than the threshold, it is determined that it is 
impossible to detect the corrected position of the 
candidate point (step 282) and the firstly given 
candidate point is outputted (step 28p). If the con- 
trast is higher than the threshold, the coordinate of 
the candidate point which is determined by the 
highest contrast is outputted as a new candidate 
point (28o). 

In Fig. 8C, determination whether in how many 
colors of three colors the corrected position of the 
candidate point can be detected is made in accor- 
dance with this evaluation (step 28q). If no position 
can not be detected, it is determined that detection 
of the corrected position of the candidate point is 
impossible (step 35) and the firstly given candidate 
point is outputted (step 2ax). If the candidate points 
can be detected in some colors, determination is 
made whether or not the corrected position of the 
candidate point determined in each color is in the 
range of a predetermined length AL (step 28r). If ail 
of them are in the range of a predetermined length 
AL, an average coordinate of the corrected posi- 
tions of the candidate points determined in these 
colors is outputted as a new candidate point (step 
28s). On the other hand, if the corrected position of 
the candidate point determined in each color is not 
in the range of a predetermined length ^AL. the 
magnitude of contrast (rate of change in light inten- 
sity) which is a measure for detecting the corrected 
position of the candidate point for each detected 



color is determined (step 28t). Determination is 
made whether or not the highest contrast is higher 
than threshold representative of a predetermined 
magnitude of the rate of change in light intensity 

5 (steps 28u, 28v). If it is lower than the threshold, it 
is determined that it is impossible to detect the 
corrected position of the candidate point (step 35) 
and the firstly given candidate point is outputted 
(step 28x). If the contrast is higher than the thresh- 

70 old. the coordinate of the candidate point which is 
detected by the magnitude of the contrast in each 
color is weighted to determine an average coordi- 
nate and is outputted as a new candidate point 
(28w). 

75 Figs. 9A to 90. 10A to IOC. 11A to 1lC, 12A to 

120, 13A to 13C show a method of operating the 
image display apparatus in Figs. 1 and 2. 

In Figs. 9A to 130. reference numeral 321 
denotes a subject image which is an automobile in 

20 this embodiment: 320 denotes a cursor for indicat- 
ing the pointing device 13 on the screen of the 
image display apparatus 11; 322 a pop-up type 
operation menu; 322a a button for starting the 
detection of the contour separating positions; 322b 

25 a button for designating termination of inputting of 
a coarse candidate point; 322c a button for des- 
ignating the cancellation of inputting of coarse can- 
didate point; 322d a button for designating the 
calculation of the candidate point of the separating 

30 point; 322e a button for preserving the coordinate 
of the candidate point; 322f a button for displaying 
the coordinate of the candidate point which has 
been preliminarily preserved; 322g a button for 
terminating the detection of the contour separating 

35 points; and 323 a display screen. 

The subject image 321 is firstly displayed in 
the display screen 323 and a selection item 322a 
in the operation menu shown in Fig. 9A is selected 
with the cursor 320 (start). The operation menu 322 

40 is displayed on the screen by outputting a menu 
display request signal to the image processing 
apparatus 10 from the pointing device 13. Selection 
of the menu is achieved by outputting an input 
request signal into the image processing apparatus 

45 10 from the pointing device 13. The image pro- 
cessing apparatus 10 accepts the coordinate of the 
cursor 320 for executing an operation correspond- 
ing to the selected item in accordance with the 
coordinate. In order to avoid a wrong operation at 

50 this time, an input request signal from the selection 
item of unnecessary operation menu is not ac- 
cepted and simultaneously, a fact that an input by 
the pointing device is not accepted is indicated to 
users by changing the colors of the selection item. 

55 In Figs. 9A to 90, the items 322b. 322d and 322e 
are displayed in different colors (as represented by 
hatching) to indicate rejection of input. The hatched 
areas are moved as the processing proceeds. After 



8 



15 



EP 0 532 823 A2 



16 



selection of the item, the operation menu 322 is 
erased from the display screen 323 by the image 
processing apparatus 10. 

As shown in Figs. 9B to 10B. a candidate point 
is inputted into a position in the vicinity of the 
contour of the subject image 321 by using the 
cursor 320. Input operation is achieved by inputting 
an input request signal into the image processing 
apparatus 10 from the pointing device 13. The 
image processing apparatus accepts the coordinate 
of the cursor 320. Since a spline curve 330 passing 
the candidate points in the order of input is dis- 
played at the time of Fig. lOB. the users can be 
helped understand the coarse contour. In this case, 
candidate points are finely inputted for a portion of 
the contour of the subject image having a large 
curvature or change in curvature as represented by 
input candidate points 331, 332, 333 and 334. After 
inputting =' of a plurality of candidate -points, the 
selection item 322b in the operation menu 322 is 
selected by using the cursor 320 (end) as shown in 
Fig. IOC. After selection, the image processing 
apparatus 10 displays a closed spline curve 349 
which passes the plurality of candidate points in 
order of input together with the subject image 321 
on the display screen 323. If the position of a 
candidate point is far from the contour, the user 
selects the candidate point by using the cursor 320 
as shown in Fig. 11 A to move the candidate point 
to a desired position 340. if necessary. This selec- 
tion of a candidate point is achieved by inputting 
an input request signal into the image processing 
apparatus 10 from the pointing device 13. When 
the candidate point is in that position, the image 
processing apparatus 10 selects it, the movement 
is performed for a period of time during which the 
input request signal is continued to be outputted 
from the pointing device 13 if a subject to be 
selected exists. The image processing apparatus 
10 accepts the coordinate indicated by the cursor 
320 for displaying the candidate point in accor- 
dance with the coordinate. 

Operation shown in fig. 11B is performed to 
change the parameters of the candidate point for 
detecting the corrected position. In other words, if 
the contour is complicated in shape, the user des- 
ignates the range with a rectangular shape, in prac- 
tice with two opposite ends of a diagonal line by 
using a cursor 320 in order. to change the param- 
eters of the candidate point. Since the tyre of a 
front wheel is complicated in shape in this exam- 
ple, a diagonal line is designated in the two points 
by indicating the outer side of the tyre after indicat- 
ing the inner side of the tyre. A firstly designated 
position 341 is displayed by a gray cursor 324 and 
the subsequently designated position 342 is dis- 
played by a cursor 320. The cursor signals are 
inputted into the image processing apparatus lO 



together with a range request signal from the point- 
ing device 13. The coordinate of the firstly ac- 
cepted cursor 320 is made as an starting point of 
the diagonal line of the rectangular shape and the 

5 coordinate of the cursor 320 while the range re- 
quest signal is being outputted is made an ending 
point of the diagonal line of the rectangular shape 
so that the rectangular shape is displayed on the 
display screen 323. The pointing device 13 outputs 

70 an input signal immediately after outputting of the 
coordinates of the rectangular area. If the external 
line 343 of the range of the rectangular shape is 
displayed in a different color when the rectangular 
shape is displayed, this will help the user briefly 

75 understand the designation of the range. In the 
figure, difference in color is represented by a dot- 
ted line. 

In such a manner, candidate points are dis- 
played on and along the contour line of an auto- 

20 mobile as shown in Fig. 11B. Only one candidate 
point is selected with the cursor 325 for a point 
which is desired to be preliminarily fixed so that, 
for example, the lower end of a front window is 
fixed and the cursor 325 is moved to the position 

25 344 which is desired to be fixed as shown in Fig. 
lie. Immediately after this movement, the can- 
didate point is fixed by outputting a position fixing 
request signal from the pointing device 13 into the 
image processing apparatus 10. The image pro- 

30 cessing apparatus 10 adds the candidate point 344 
located in the coordinate of the cursor 322 with an 
attribute representing that movement in the contour 
separating processing is not performed and 
changes the color of the candidate point 344 dis- 

35 played on the display screen 323 for helping the 
user understand the designation of the position to 
be fixed. 

Then, in order to determine the separating po- 
sition of the image, the operation menu 322 is 

40 displayed and a selection item of the menu is 
selected with the cursor 320 as shown in Fig. 12A. 
The selection item 322d is a button for designating 
execution of the calculation of the separating can- 
didate point. If a problem occurs for some reason 

45 while the candidate points in Figs. 9B to 12A are 
being inputted, the operation menu 322 is dis- 
played and the inputted candidate points can be 
cancelled by designating the button 322c for can- 
cellation. In order to preserve the coordinate of the 

50 inputted candidate point, the inputted candidate 
point can be preserved by specifying the button 
322f for preserving the coordinate of the candidate 
point. The coordinate of the preserved candidate 
can be read out by specifying the button 322f for 

55 displaying the coordinate of the candidate point on 
the screen. Detection of the contour separating 
point can be terminated by the button 322g for 
terminating the detection of the contour separating 
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position. If fixing of the positions of the candidate 
points and changing of detection parameters is 
carried out as described with reference to Figs. 
11A to lie, the innage processing apparatus 10 
detects the image separating point by also using 
information obtained when the selection item 322d 
is selected. The result is displayed as a closed 
spline curve 345 passing a plurality of candidate 
points in order of input together with the subject 
image 321 as shown in Fig. 12B. In this case, a 
point whose corrected position is not detected is 
displayed in a different color for helping the user 
understand the image separating point. In lieu of 
indicating the portion which has not been detected 
in a different color, the candidate points which can 
be detected are displayed using white circles in 
Fig, 12B. After processing, candidate points in 
which the threshold for determining the image sep- 
arating points is changed can be preset for the 
candidate points (346, 347 and 348) which are not 
detected by performing the same operation as 
shown in Fig. 12C. Thereafter, processing for de- 
tecting the image separating point is carried again 
from this , state by performing the same operation 
as Fig. 12A as shown in Fig. 13A if necessary. As a 
result of this, the image separating points are a 
closed spline curve 350 which passes these points 
are displayed as shown in Fig. 13B. 

A slide bar and a digital indicator may be 
displayed together with the rectangular shape so 
that the parameters can be clearly read if the 
parameters are increased or decreased -in Figs. 
118 and 12C- The image processing apparatus 10 
displays a slide bar 360 which is shown in Fig. 
13C. The slide bar 360 indicates an increase or 
decrease in parameters of threshold which are to 
be changed in the rectangular shape 343 by the 
amount of slide. While the coordinate of the cursor 
320 reflecting the pointing device 13 on the screen 
323 matches with the coordinate of the slide bar 
360 and a movement request signal is continued to 
be outputted from the pointing device 13 to the 
image processing apparatus 10, the image pro- 
cessing apparatus 10 displays a bar 362 of the 
slide bar 360 at the same position as the cursor 
320 reflecting the coordinate of the pointing device 
13 on the screen 323. At this time, the image 
processing apparatus 10 increases or decreases 
the parameters depending upon the coordinate 
moved by the pointing device 13 and displays the 
state of the parameters on a panel 361. In this 
case, the range request signal which is outputted 
from the pointing device 13 may be the same as 
the signal for moving. In this case, determination of 
the signal is made in current input state in the 
image processing apparatus 10. 

Fig. 14 is a block diagram showing an embodi- 
ment of an image separating system of the present 



invention. In Fig. 14, reference numeral 400 de- 
notes an image separating apparatus; 401 a mod- 
ule for determining a coarse contour curve; 402 a 
module for determining the candidate point of the 

5 initial separating point: 403 a module for determin- 
ing contour candidate points based upon the red 
light spectrum; 404 a module for determining con- 
tour candidate points based upon the green light 
spectrum; 405 a module for determining contour 

10 candidate points based upon the blue light spec- 
trum; 406 a module for comparing the contour 
candidate points determined based upon each col- 
or light spectrum with each other; 408 a module for 
separating an image from the specified separating 

75 points; 409 a memory for holding an original image 
which is an object to be separated; 41 0 a pointing 
device; and 41 1 a memory for holding a separated 
image. 

A coarse contour curve and candidate points of 

20 an initial separation points are determined by the 
module 401 for determining a coarse contour curve 
with reference to an original image including a 
subject to be separated which is provided from the 
memory 409 and the module 402 for determining 

25 the candidate points of an initial separating points. 
The contour - curve and the separating candidate 
points are determined by inputting the coarse con- 
tour curve in an interactive manner by a user if the 
subject to be separated is a still image. If the 

30 subject has a distinctive color, a coarse contour of 
areas of the color is determined. In the former 
method, a coarse contour curve or the candidate 
points can be inputted by using the pointing device 
410 with reference to, for example, the original 

35 image of the memory 409. The latter method can 
be easily carried out by using, for example, a 
method which is disclosed in JP-B-2-4027, for ex- 
ample. 

If the subject is a motion image, a coarse area 

40 of a changed or moved subject can be obtained by 
detecting the difference between two frames which 
are continuous in time from successive still images 
(each of which is referred to as frame) at intervals 
of a given unit time. This is possible since the 

45 background portion can be regarded as hardly 
changed in a short interval between frames. A 
coarse area of the motion image can be easily 
determined, for example, by a method disclosed in 
the above described article "Development of video 

50 surveillance system using motion information". The 
points on the boundary of the determined coarse 
area are made the initial separating candidate 
points. On the basis of the initial separating can- 
didate points and an original image of the memory 

55 409 which is in practice a subject of the contour 
separation, accurate separating candidate pomt in 
each spectral component is determined from each 
of the modules 403. 404 and 405 for determining 

10 
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the contour candidate points of respective spectral 
components. This is due to a fact that when the 
boundary between a yellow area and a green area 
which have the same brightness is to be deter- 
mined, the red light spectral component which is 5 
not included in the green light spectral component, 
but is included the yellow spectral component can 
be easily detected by using the module 403 for 
determining the contour candidate points based 
upon the red fight spectrum. The contour candidate io 
points determined in respective spectral compo- 
nent are compared with each other by the module 

406. This is because that when the boundary be- 
tween a yellow area and a green area having the 
same brightness is determined, the green spectral 75 
component included in both colors can be detected 

by using the module 404 for determining the con- 
tour candidate points based upon the green light 
spectrum white the contour candidate points cannot 
be detected by using only the module 404 since 20 
the green spectral components included in both 
yellow and green spectra are same in light inten- 
sity. Accordingly, contour candidate points which 
are closer to a true contour can be calculated by 
using the detectable red spectral component. On 25 
the basis of the results from the modules 403. 404 
and 405. contour candidate points which are closer 
to a true contour can be determined as separating 
points by the separating point specifying module 

407. The module 408 for separating an image 30 
determines each of the coordinates on the external 

line of the image to be separated by interpolating 
the image separating points determined by the 
separating point specifying module 407 which are a 
set of the separating points with a straight tine or a 35 
smooth curve. An image inside or outside of the 
external line is separated from the original image 
stored in the memory 409 based upon these co- 
ordinates. The separation can be easily achieved 
by using a method which is disclosed in detail in 40 
the above cited article "Scanline Conversion of a 
polygon (2)" (PIXEL NO. 16. pages 136 - 141). The 
separated image is preserved in the memory 41 1 
and is outputted from the image separating appara- 
tus 400. 45 

When it is not necessary to preserve the sepa- 
rated image in the image separating apparatus 400, 
an output of the module 408 may be directly out- 
putted externally of the apparatus 400 without stor- 
ing it in the memory 411. so 

The present embodiment is an examplarly 
structure for performing the image separating point 
detection and image separating processing. It is to 
be understood that the invention is not limited to 
only this structure. 55 

Candidate points on the contour ctoser to a 
true image are determined by specifying an coarse 
contour and detection of the image separating 



points is carried out and the image separating 
processing is then carried out in accordance with 
the present embodiment. 

As mentioned above, only the edge of a given 
subject or object can be positively determined by 
using color components in an image in accordance 
with the present invention. Since a user can di- 
rectly intervene in a portion having such a com- 
plicated shape or a small change in light intensity 
that the edge is difficult to determine on a screen 
by a simple operation, the separation points of an 
image can be detected at a high precision while 
automatic process for detecting image separating 
point is being performed. 

Claims 

1. A method for detecting image separating 
points where only a given subject image is 
separated from images, comprising the steps 
of: 

a first step (21) for displaying on a screen 
a plurality of candidate points in the vicinity of 
a boundary of said given subject image, which 
are inputted by a user; 

a second step (26) for determining and 
displaying a smooth curve which connects the 
plurality of candidate points: 

a third step (27) for determining intensity 
of light along a straight line normal to said 
curve at each of the plurality of the candidate 
points; 

a fourth step (32) for correcting the posi- 
tion of said candidate point based upon the 
rate of change in the intensity of light deter- 
mined in each of the candidate points; and 

a fifth step (33) for determining and dis- 
playing a smooth curve passing said corrected 
candidate points. 

2. A process for detecting image separating 
points according to Claim 1, further including 
the steps of: 

individually processing the same image in 
each of components having different wave 
lengths by using the third and fourth steps; 

determining the positions of the corrected 
candidate points for each component: 

calculating the position of one corrected 
candidate point from a plurality of corrected 
candidate points after processing of said fourth 
step; 

performing the sixth step for determining . 
the final position; and 

thereafter performing said fifth step. 

3. A method for detecting image separating 
points according to Claim 1 or 2 in which when 
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the rate of change in the intensity of light is 
determined in said fourth step, the designated 
values of the threshold parameters for deter- 
mining the rate of change for detecting an 
edge are changed. 

4. A method for detecting image separating 
points according to Claim 1 or 2 in which when 
a smooth curve connecting the plurality of can- 
didate points is determined in the second step, 
the position of at least one of the candidate 
points is preliminarily fixed and processing in 
said third and fourth steps is stopped for the 
fixed candidate point, while processing of the 
second to fifth steps is performed for the other 
candidate points. 

5. A method for detecting image separating 
points according to Claim 1 or 2 in which, in 
the fourth stop, processing for determining the 
rate of change in the intensity of light is se- 
quentially performed from the external to inter- 
nal slide at each candidate point on a coarse 
curve traced in the vicinity of the contour of 
the subject and a point which satisfies con- 
ditional formulae (Eqs. (1), (2) and (3)) repre- 
senting that the rate of change for correcting 
said plurality of candidate points is higher than 
a threshold is determined as the position of the 

. corrected candidate point. 

6. A method for detecting image separating 
points according to Claim 5 in which after 
processing of said fifth step, said threshold of 
the rate of change is changed for at least one 
of candidate points which are determined as 
not satisfying said conditional formulae as a 
result of the determination of the rate of 
change in light intensity in said fourth step, 
and then processing of said third to fifth steps 
is repeated at least once for said at least one 
candidate point. 

7. A method for detecting image separating 
points according to Claim 5, further including 
determining a candidate point inputted in the 
first step to be the position of the corrected 
candidate point; and adding to said candidate 
point an attribute representing that it does not 
satisfy the conditional formulae for correcting 
the position, including displaying said candi- 
date point in a color different from that of the 
other point on the screen. 

8. A method for detecting image separating 
points according to Claim 5, further including 
determining a candidate point inputted in the 
fourth step to be the position of the corrected 



candidate point: and adding to said candidate 
point an attribute representing that it does not 
satisfy the conditional formulae for correcting 
the position, including displaying said candi- 
5 date point in a color different from that of the 

other point on the screen. 

9. A method for detecting image separating 
points according to Claim 1 or 2 in which if 

10 processing is repeated again from the second 

step after completion of said fifth step, at least 
one candidate point is fixed and the third and 
fourth steps are stopped for the fixed can- 
didate point and each processing of the third. 

75 fourth and fifth steps is repeated at least once 

from a point adjacent to the fixed point for the 
other candidate points. 

10. An image separating apparatus for separating 
20 a given subject image from images, compris- 
ing: 

means for determining a coarse contour 
curve for the given subject image which is 
desired to be separated: 
25 means for determining a plurality of can- 

didate points on said contour curve: 

means for correcting the position of each 
of said candidate points in each spectral com- 
ponent based upon the rate of change in light 
30 intensity of each spectral component: 

means for comparing the candidate points 
corrected in each spectral component with 
each other; 

means for specifying the separating points 
35 from the results of comparing; and 

means for separating the image from the 
specified points. 



40 



45 



50 



12 



EP 0 532 823 A2 



FIG. I 



12 



15 



IMAGE 
RECORDING 
APPARATUS! 



14 

± 



IMAGE 
MEMORY 



SEPARATING 
MEMORY 



IMAGE PROCESSING 
APPARATUS 



T 

10 



IMAGE 
DISPLAY 
APPARATUS 



POINTING 
DEVICE 



F IG. 2 

18 



If 


17 


( ( 


IMAGE 


CONTOUR 


MEMORY 


MEMORY 



16 
-t 



SEPARATING 
MEMORY 



IMAGE 
RECORDING 
APPARATUS 



IMAGE PROCESSING 
APPARATUS 

\ 

10 



IMAGE 
DISPLAY 
APPARATUS 

^ 

POINTING 
DEVICE 



>3 



EP 0 532 823 A2 



FIG. 3 



20 



START 



3 



INPUT CANDIDATE POINTS Pi OF GIVEN IMAGE a IN THE 
WHOLE OF IMAGE A AND DISPLAY THE SAME ON DISPLAY APPARATUS 



26- 



DETERMINE SPLINE CURVE Ck PASSING CANDIDATE POINlii Ki 
AND DISPLAY THE SAME ON DISPLAY APPARATUS 



22-^ DESIGNATION OF CANDIDATE POINT WHICH IS NOT 



moved is requi red ? 
Tyes" 




23- 



DESIGNATE CANDIDATE POINT Pi V^HICH IS NOT MOVED 



24 

25- 
3 

27- 
32- 

28- 

33- 
29 




DESIGNATION OF CANDIDATE POINT FOR WHICH 

threshold for det ecting edge is changed is required ? 

Tyes" 



NO 



DESIGNATE CANDIDATE POINT Pi FOR WHICH THRESHOLD 
FOR DETECTING EDGE IS CHANGED 




I 



MAY CANDIDATE POINT BE MOVED ? 



NO 



»YES 



DETERMINE LIGHT INTENSITY IN A DIRELTION OF 
NORMAL LINE PASSING CANDIDATE POINT Pi AND 
DETERMINE THE DIFFERENCE BETWEEN DETERMINED 
VALUE AND A VALUE IN THE VICINITY THEREOF 



DETERMINE THE COORDINATE OF EDGE DETECTION POINT 
FOR WHICH THE DIFFERENCE EXCEEDS ARE DETERMINED 
THRESHOLD AND MOVE CANDIDATE POINT TO THIS 

COORDINATE 



IN CASE OF IMAGE INCLUDING A PLURALITY OF 
DIFFERENT WAVE LENGTH COMPONENTS, CORRECT 
CANDIDATE POINT DETERMINED FOR EACH COMPONENT 
AND DETERMINE FINAL CANDIDATE POINT 



DETERMINE SPLINE CURVE Ck' PASSING NEW CANDIDATE 
POINT Pi AND DISPLAY IT ON DISPLAY APPARATUS 




TERMINATION OF PROCESSING 




NO 



30' 



^YES 
END 



EP 0 532 823 A2 



FIG. 4A 




FIG. 4C 



FIG. 4D 



EP 0 532 823 A2 



FIG. 5A 




FIG. 5B 




16 



EP 0 532 823 A2 



FIG. 5C 




f 




RED LIGHT INTENSITY 


206 


GREEN LIGHTT ^ 

INTENSITY ^ >V 




BLUE LIGHT /I ' \ | 
INTENSITY / I \ 1 


207 

— -^.^^^^/^208 


Eb A Eg E:r 
Pl\2l4 


NORMAL VECTOR 





17 



EP 0 532 823 A2 



FIG. 6A 



RED LIGHT 
INTENSITY 




FIG. 6B 



223 




EP 0 532 823 A2 

FIG. 7A 



RED LIGHT INTENSITY 




FIG. 7B 




NORMAL 
VECTOR 

p^238 



19 



EP 0 532 823 A2 




20 



EP 0 532 823 A2 



00- 



00 



E 

CO 
CsJ 





















2 z 


1 — ^ 










a: 




h- 




LU 

Q 






Q 

LU 
h- 

O 

LU 
UJ 

Q 



00 cn 

O VD 




m 

00 



00 



Ouj 
Q-00 

=3 

< 



cn 

LlI 

>- 




li- 



21 



EP 0 532 823 A2 



00 
(M 



in. 



9P 

CVJ 



r3 
GO 
OJ 



> 

00 
C\J 








Q 






-J 






O 


1— 




X 


00 




00 


< 




LU 


en 




cc 


1- 00 






2 h- 




H- 


O 2 




A 


^ O 




LU 






^ c 






ETERM 




ITRAST 
POINTS 


o 




O c 
u 



CD 

S o 
o ol 

uu — 
O ^ 

ei; CO 
t— o 

LU a. 
O 




00 Q 

cr 



00 O 

o 



V 



00 



LU 

Q 



X < 



UJ 



I" 

LU 
>- I- 

CD UJ 

00 00 
< UJ 



Q 

cc 

o 
o 
o 



Q 

LU 
f— 

X 

LU 



O Q 
T O 



o 
u 

o 

Z 

> 
< 
X 

cr 
O 
_i 

o 
u 



o 

X 

CO 

LU 
CC 
LU 



< 

I- 
ir 

LU 

X 
X 



00 
(M 



to 
00- 
(M 



UjUJyj 

<o c 
o 
o 



o 

CD 



22 



EP 0 532 823 A2 



FIG. 9A 




FIG. 9B 



323" 




FIG. 9C 



323 




23 



EP 0 532 823 A2 



FIG. lOA 



323 




FIG. lOB 



323 




FIG. IOC 

323- 



32 



END 



349 



322 

} ^322a 

-320 
--322b 
^322c 
^322d 
^322e 
^322 f 
^322g 



RESET— 



QUIT 



OA 




25 



EP 0 532 823 A2 



FIG. I2A 



323 




FIG. I2B 



323- 



FIG. I2C 



323 



322. 

/ . -322a 
^2:^^322b 



322c 
cowT0ui?w ^^22d 

^ 322e 





521 
In 7., 






u 






320 



26 



EP 0 532 823 A2 



FIG. 13 A 



323- 




322 
f— ^522q 

STARTr '>^322b 
~^]'^^322c 

L0ADXJ--322e 
ourrkS322f 
^322g 




FIG. I3C 

360 



27 



EP 0 532 823 A2 



FIG. 14 



,400 



MEMORY FOR 
HOLDING ORIGINAL 
IMAGE TO BE 
SEPARATED 



.409 



410- 



POINTING 




DEVICE 





401 



MODULE FOR 
DETERMINING 
COARSE CURVE 



402- 




MODULE FOR 
DETERMINING 
INITIAL SEPARATING 
CANDIDATE 
POINTS 



MODULE FOR 
DETERMINING 
CONTOUR 
CANDIDATE 
POINTS BASED ON 
RED LIGHT 
SPECTRUM 



MODULE FOR 
DETERMINING 
CONTOUR 
CANDIDATE POINTS 
BASED ON 
GREEN LIGHT 
SPECTRUM 



404 



MODULE FOR 
CONTOUR CAN! 


f 

COMPARING 
DIDATE POINTS 


\ 




MODULE FOR DESIGNATING 
SEPARATING POINT 






IMAGE SEPARATING MODULE 


\ 




MEMORY FC 
SEPARATE 


)r holding 
:d image 



MODULE FOR 
DETERMINING 
CONTOUR 
CANDIDATE 
POINTS BASED ON 
BLUE LIGHT 
SPECTRUM 

405 



.406 



,407 



.408 



9fl 



J) 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 




© Publication number: 



0 532 823 A3 



® 



EUROPEAN PATENT APPLICATION 



© Application number: 92102774.4 
@ Date of filing: 19.02.92 



© Int. ci.S: G06F 15/68 



(S) Priority: 18.09.91 JP 237899/91 


^71) Annll/^ant* INQTITIITC COO DCDCOkiai i^cr\ 


Date of Dublication of annllfstlnn* 


IMPORMATtOKf CKI\/IDOKlh/iC^KtT* 
irMr^jniVlA 1 lUN VmVJIMivlcNT 


1 oranornon- 1 •criorne 


24 03 93 Bulletin 

fc*t«wO>wO wU lie till 90/ 1 *B 


iviinaio*KU, 1 OKyo iOo(JP) 


(so Desionated Contractino States* 


invenior. iKcZowa, iviiisuru 


DE FR GB 


31-9-408, Josuihoncho-1-chome 




r\.wuaira*sni^ jr'/ 


(s9) Datp of ripfprrpri ni ihlir'atirin nf thp QParr-h r^nnrt- 


Inventor: Miyatake, Takafumi, Hitachi 


05 01 94 Bulletin 94/01 


Fxoyasuuai Apaio a^U4 




32, Koyasumachl-2-chome 




Hachioji-shi(JP) 




Inventor: Ueda, Hlrotada 




22-49, Nishimachi-2-chome 




Kokubunji-shi(JP) 




Inventor: Sumino, Shigeo 




Towa Sengawa Kopu 202, 




27-1, Midorigaoka-2-chonne 




Chofu-shi(JP) 




Inventor: Yoshizawa, Satoshi 




1755-91. Ozenji, 




Asao-ku 




Kawasaki-shi(JP) 




© Representative: Strehl Schubel-Hopf Greening 




& Partner 




Maximilianstrasse 54 




D-80538 Munchen (DE) 



CO 

< 

CO 
CM 
00 

C4 

CO 

m 



o. 

LU 



© Method and apparatus for detecting the contour and separating a given subject from an image. 



© A method for detecting a countour of a given 
subject to be separated from images and an image 
separating system. First, a plurality of candidate 
points (110.-120) which have been input by a user 
are displayed on a screen. A smooth curve (100) 
which connects the plurality of candidate points is 
determined and is displayed on a display apparatus 
(Fig. 4A). The intensity of light on a straight line 



(130) extending in a direction normal to each can- 
didate point is determined (Fig. 48). The position of 
the candidate point is corrected based upon the rate 
of change in the intensity of light which is deter- 
mined for each candidate point. A smooth curve 
(101, 102) which passes the corrected points is 
determined and is displayed on the display appara- 
tus (Figs. 4C, 4D). 



Rank Xerox (UK> Business Services- 



EP 0 532 823 A3 



FIG. 4A 




FIG. 4C FIG. 4D 




2 



Kuropcan Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 92 10 2774 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



X 
A 



X 
A 



CitMtion of document with indication, t^hcrc appropriate, 
of relevant pass JSCS 



EP-A-0 334 230 (FUJI PHOTO FILM CO., 
LTD,) 

* abstract * 

* column 16, line 13 - line 39 * 

* column 17. line 9 - line 13 * 

* column 17. line 52 - column 18, line 8 * 

US-A-4 961 425 (KENNEDY ET AL.) 

* abstract • 

* column 5. line 32 - line 40 * 

* column 6. line 5 - line 40 " 

EP-A-0 342 379 (FUJI PHOTO FILM CO., LTD.) 

* abstract * 

* column 6, line 25 - line 53 * 

* column 7, line 26 - line 54 * 

* column 9. line 10 - line 34 ^ 

IEEE TRANSACTIONS ON MEDICAL IMAGING 
vol. 7. no. 2 . June 1988 , NEW YORK, USA 
pages 81 - 90 

C. HENRY CHU ET AL . 'Detecting Left 
Ventricular Endocardial and Epicardial 
Boundaries by Digital Two-Dimensional 
Echocardiography. ' 

* page 83. left column,, line 51 - line 55 

* page 85, right column, line 54 - line 58 

* page 86, rignl column, line 45 - line 55 



cvGiP: Image understanding 

vol . 53. no. 2 , March 1991 , USA 
pages 211 - 218 

COHEN L D 'On activr contour models and 
bal loons . ' 



The present search report hus* been drut^n up fur ull clittms 

rtjcc of tcvck 



THE HAGUE 



[i^r ot iOtr%fl€ttom of Ibc wwcb 

3 November 1993 



Reloant 
to ctaim 



1.3.9 
2,4-8,10 



1.3,5 
2,4,6-10 



1-10 



OjlSSIHCAnON OK TIIE 
APPIJCATION (IntCtS) 



G06F15/68 



1.3-9 



tkc:H:nk;ai. hkij)S 

SKARCtlKU flni,Cl-5) 



G06F 



GONZALEZ. ORDONEZ. 0 



<:AlKt;tlKV UK<:MK1) DtKl-MKMS 

X : particularly relevant if taken alune 

y : particularly relevant if cnmbmed «ith anuther 

(document of th« same catci£or\ 
A : technotoi£ical bacLtfround 
U : non-written disdusurc ' 
P : tMemiedialc document 



: theory or principle underlying the tnvefliion 
: earher patent document, but published on, or 

after the filing date 
: document dted in the application 

document dted for other reasons 



& : member of the same patent family, cnrrespoodins 
document 



THIS PAGE BLAMKl"*™ 



I 



